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Chapter  1:  Introduction  to  Peer  Review 


The  assumption  underlying  the  value  of  peer  review  is  that  the  quality  of  work  is 
substantiated  or  improved  through  critiques  by  individuals  who  are  independent, 
objective,  and  have  specialized  knowledge  in  the  subject  matter.  One  of  the  authors  (Dr. 
Lyons)  has  had  firsthand  experience  with  such  reviews.  Some  time  ago,  while  in  the 
private  sector,  he  was  asked  to  serve  on  a  National  Research  Council  (NRC)  ad  hoc 
committee  of  external,  independent  experts  to  review  the  fire  research  program  at  the 
National  Bureau  of  Standards  (NBS).  The  committee  found  the  programs  to  be 
fragmented  and  deficient  in  basic  research,  and  made  a  number  of  recommendations  on 
how  to  improve  them.  (For  more  on  how  such  a  committee  operates,  see  the  discussion  of 
the  National  Institute  of  Standards  and  Technology  in  chapter  3.)  The  result  was  the 
creation  of  a  newly  focused  organization  with  a  strengthened  scientific  knowledge  base. 
These  reforms  helped  NBS  become  a  world  leader  in  the  field  of  fire  research. 

A  second  example  of  the  contributions  made  through  the  peer  review  process  is  provided 
by  the  establishment  in  1995  at  the  Army  Research  Laboratory  (ARL)  of  an  NRC  board, 
the  Technical  Assessment  Board,  to  review  the  quality  of  ARL  programs.  The  first  report 
made  a  number  of  recommendations,  including  reduction  of  what  the  panel  called 
parochialism  at  ARL.  Actions  taken  as  a  result  of  this  review  improved  the  laboratory’s 
interactions  with  academia  and  industry.  Such  outside  review  often  provides  a  different 
perspective  that  can  be  useful  in  shaping  future  work.  The  use  of  peer  review,  using 
experts  in  possession  of  varied  and  extensive  experience,  can  much  improve  the  quality 
of  work  and  efficacy  of  results. 

The  Army  Science  and  Technology  Executive,  Dr.  Thomas  Killion,  requested  a  study  of 
peer  review  methods  in  use  at  Army  laboratories.  This  paper  discusses  Army  laboratories 
in  tenns  of  generally  accepted  best  practices  and  compares  them  with  techniques  at  other 
DOD  laboratories,  and  at  a  few  other  government  agencies. 

Background 

Peer  review  of  technical  work  is  a  well-established  tradition  in  research  and  development. 
Review  can  be  on  individual  projects  or,  more  broadly,  of  a  laboratory  as  a  whole.  There 
are  three  forms  of  project- focused  review:  one  is  used  to  evaluate  proposals  for  new 
work,  such  as  research  proposals  to  funding  agencies;  a  second  is  used  to  review  work  in 
progress,  which  is  the  primary  focus  of  this  study;  and  the  third  is  used  to  review 
completed  work  in  the  form  of  reports  or  publications.  In  peer-reviewed  journals, 
sometimes  called  archival  journals,  each  paper  is  sent  to  technical  experts  for  critique; 
changes  may  be  suggested,  or  the  paper  may  be  rejected,  based  on  lack  of  technical  merit 
or  originality. 

While  peer  review  has  been  used  for  many  years  in  the  Federal  laboratories,  there  are  no 
uniform  standards  or  approaches.  In  a  1983  report  to  the  White  House  Science  Council, 
its  Federal  Laboratory  Review  Panel  recommended  that  individual  projects  be  subject  to 
peer  review  at  the  outset.  The  panel  also  recommended  the  creation  of,  in  its  own  words, 
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“an  oversight  function  responsible  for  assuring  the  continuing  excellence  of  the 
laboratory.  This  function  could  be  perfonned  by  an  external  committee  [and]  would  focus 
on  productivity  and  on  the  excellence,  relevance,  and  appropriateness  of  research.”1 

The  report  provided  no  details  as  to  how  the  reviews  should  be  performed.  The  President 
instructed  the  Office  of  Science  and  Technology  Policy  and  the  Office  of  Management 
and  Budget  to  lead  an  interagency  effort  to  respond  to  the  Panel’s  recommendations.  The 
results  of  this  effort  are  documented  in  a  1984  report.2  Various  actions  were  undertaken; 
in  particular,  the  DOD  asked  the  Services  to  form  external  advisory  groups. 


Peer  review  was  invented  at  the  Royal  Society  in  1665  by  the  founding  editor  of 
Philosophical  Transactions  of  the  Royal  Society,  Henry  Oldenburg.  The  present-day 
peer  review  system  evolved  from  this  17th-century  process.” 

A  practice  similar  to  a  peer  review  process  is  found  in  Ethics  of  the  Physician,  written 
by  Ishaq  bin  Ali  al-Rahwi  (854-931)  of  al-Raha,  in  what  is  now  Syria.  His  work,  as 
well  as  later  Arabic  medical  manuals,  state  that  a  visiting  physician  must  always  make 
duplicate  notes  of  a  patient's  condition  on  every  visit.  When  the  patient  was  cured  or 
had  died,  the  notes  of  the  physician  were  examined  by  a  local  medical  council  of  other 
physicians,  who  would  review  the  practicing  physician's  notes  to  decide  whether  his/her 
performance  have  met  the  required  standards  of  medical  care.  If  their  reviews  were 
negative,  the  practicing  physician  could  face  a  lawsuit  from  a  maltreated  patient.  “ 

Peer  review  has  been  considered  essential  to  the  modem  scientific  method  only  since 
the  middle  of  the  20th  century.  Before  then,  its  application  was  frequently  lax.  For 
example,  Albert  Einstein's  revolutionary  "Annus  Mirabilis"  papers  in  the  1905  issue  of 
Annalen  der  Physik  were  not  reviewed  by  anyone  other  than  the  journal's  editor  in  chief, 
Max  Planck  (the  father  of  quantum  theory),  and  its  co-editor,  Wilhelm  Wien.  Although 
clearly  they  were  peers  (both  won  Nobel  prizes  in  physics),  a  formal  panel  of  reviewers 
was  not  sought,  as  is  done  for  many  scientific  journals  today.  Established  authors  and 
editors  were  given  more  latitude  in  their  journalistic  discretion  at  that  time.  As  a  recent 
editorial  in  Nature  stated,  "in  journals  in  those  days,  the  burden  of  proof  was  generally 
on  the  opponents  rather  than  the  proponents  of  new  ideas.” 

Source:  derived  from  Wikipedia  entry  on  peer  review,  available  at  <http://en.wikipedia.org/ 
wiki/peer_review>. 


1  Office  of  Science  and  Technology  Policy,  Report  of  the  White  House  Science  Council  Federal  Laboratory’ 
Review  Panel,  Recommendation  4-1  (Washington,  DC:  Executive  Office  of  the  President,  1983). 

2  Office  of  Science  and  Technology  Policy,  Progress  Report  on  Implementing  the  Recommendations  of  the 
White  House  Science  Council ’s  Federal  Laboratory  Review  Panel  (Washington,  D.C:  Executive  Office  of 
the  President,  1984). 
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Peer  review  tests  the  effectiveness  of  a  laboratory’s  research  staff  by  comparing  the 
conduct  of  the  laboratory’s  research  with  that  at  the  best  laboratories  in  the  field.  Outside 
experts  compare  the  material  under  review  with  what  they  know  of  comparable  work 
elsewhere — perhaps  including  their  own  work.  External  reviews  reduce  the  tendency  of  a 
laboratory  to  become  parochial,  a  special  problem  for  laboratories  that  perform  classified 
research.  The  expectation  of  scheduled  peer  reviews  challenges  the  laboratory’s  technical 
staff  to  keep  up  with  technical  advances  worldwide  and  to  stay  in  touch  with  colleagues 
elsewhere.  Peer  review  also  brings  varying  perspectives  on  the  work  from  laboratories 
across  the  Nation,  or  even  around  the  world.  Therefore,  peer  review  is  a  way  to  establish 
a  reputation  for  high-quality  performance  at  a  laboratory.  For  some  portion  of  the 
assessment,  review  may  be  enhanced  by  the  provision  of  data  on  performance  against  a 
number  of  metrics.3 

Some  Factors  in  Laboratory  Quality 

There  are  several  factors  in  evaluating  the  quality  of  individual  programs  or  groups  of 
programs  in  a  laboratory: 

Status  of  the  work.  Planning,  originality  and  creativity,  publications  and  patents, 
comparison  to  work  being  done  elsewhere,  coordination  or  collaboration  with  others,  and 
pacing  or  productivity  are  considered.  The  panel  may  also  assess  the  program  as  a  whole, 
e.g.,  the  balance  among  basic,  applied,  and  development  research.  Within  these 
categories,  reviewers  may  also  consider  the  potential,  or  lack  thereof,  for  beginning  new 
areas  of  basic  research.4 

Qualifications  of  the  research  team.  Reviews  look  at  the  record  of  accomplishments, 
awards  and  honors,  and  ability  to  present  the  work.  First-rate  equipment  and  modem 
facilities  are  essential  for  success — are  they  well  supported?  Is  the  overall  budget 
adequate  for  the  task?  Finally,  the  working  environment  is  important  to  an  evaluation. 
Stimulating  coworkers  with  technical  prowess  in  their  fields,  and  a  supportive 
management  team  are  important  elements  of  productive  laboratory  work. 

Relevance.  This  paper  is  not  focused  on  relevance,  though  many  of  the  reviews  discussed 
herein  address  a  mixture  of  quality  and  relevance.  Factors  in  assessing  relevance  include: 
probability  of  success,  priority  of  the  need  being  addressed,  validation  of  the  need  by  the 
user,  and  collaborations  with  other  laboratories.  Relevance  is  also  influenced  by  the 
timeliness  of  the  delivery  of  results. 

Credibility.  For  a  review  to  have  credibility  across  communities  of  interest,  reviewers 
must  be  independent  and  impartial,  able  to  comprehend  technical  details,  and  familiar 
with  similar  work  being  done  elsewhere.  Reviewers  can  be  drawn  from  within  the 
institution,  if  they  are  appropriately  remote  from  the  work,  or  from  outside  sources  such 


3  Edward  A.  Brown,  Reinventing  Government  Research  and  Development:  A  Status  Report  on  Management 
Initiatives  and  Reinvention  Efforts  at  the  Army  Research  Laboratory,  ARL-SR-57  (Adelphi,  MD:  Army 
Research  Laboratory,  August  1998). 

4  Hans  Mark  and  Arnold  Levine,  The  Management  of  Research  Institutions:  A  look  at  government 
laboratories,  NASA  SP-481  (Washington,  DC:  NASA,  1984),  226. 
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as  academia,  industry,  or  other  institutions.  Any  potential  for  bias  or  the  perception  of 
bias  must  be  resolved. 

Reporting.  Review  panels  should  make  explicit  reports,  preferably  in  writing,  to  the  head 
of  the  laboratory,  and  perhaps  to  higher  levels  of  external  management.  The  personnel 
reviewed  should  be  given  the  results  of  the  review.  It  is  unusual  to  find  a  laboratory  that 
cannot  benefit  from  a  thoughtful,  professional  critique.  Such  outside  reviews  should  be 
sought  by  practitioners  in  any  field.  For  laboratories,  the  review  may  take  many  forms. 
Some  of  these  fonns  are  investigated  in  this  paper  in  terms  of  current  practices  at  various 
Army,  DOD,  and  other  government  laboratories. 

Managing  the  Peer  Review  Process 

The  peer  review  process  is  generally  strengthened  by  attention  to  the  following  concerns: 

Review  process.  Two  archetypal  methods  of  establishing  a  reviewing  system  should  be 
discussed.  In  the  first,  the  laboratory  under  review  may  contract  for  the  review  with  an 
outside  party  and  only  make  recommendations  to  the  contractor.  In  this  model,  the 
contractor  manages  the  selection  of  panel  members  and  provides  logistical  and 
operational  support  to  the  review  panel.  After  finalizing  the  contract,  the  laboratory  plays 
only  a  supporting  role — offering  suggestions,  but  otherwise  maintaining  a  hands-off 
approach.  In  the  contrary  model,  the  laboratory  selects  panel  members,  defines  review 
topics,  and  exercises  close  management  control — a  hands-on  approach.  The  advantage  of 
the  hands-off  approach  is  the  enhanced  credibility  of  the  product.  An  advantage  of  the 
hands-on  approach  is  that  management  is  able  to  select  panel  members  known  to  be 
particularly  expert  in  the  details  of  the  programs  under  review. 

Relationship  of  reviewers  to  reviewed.  Another  important  factor  is  the  relationship  of 
the  members  of  the  review  panel  to  the  laboratory  under  review.  To  establish  credibility, 
panel  members  should  be  seen  as  objective  and  independent  of  the  laboratory.  To  this 
end,  experts  should  be  selected  from  outside  the  laboratory,  and  preferably  outside  the 
organizational  parent  of  the  laboratory.  The  use  of  panels  selected  from  internal  staff 
gives  the  appearance  of  favoring  the  laboratory’s  efforts  out  of  loyalty  and  parochialism. 
Review  panel  members  should  include  academics  qualified  to  look  at  the  basic  research 
component.  For  applied  research,  members  should  be  added  from  industry  and  other 
relevant  laboratories.  Possible  conflicts  of  interest  should  be  made  known,  and  the  panel 
constituted  with  appropriate  balance  to  offset  conflicts.  Another  benefit  of  external 
review  is  the  injection  of  different  experiences  and  points  of  view  not  available  to  in- 
house  staff. 

Quality  and  relevance.  Quality  is  best  judged  by  expert  scientists  and  engineers  active  in 
the  appropriate  fields.  Relevance  is  best  judged  by  the  users  of  the  technical  results,  who 
may  or  may  not  know  the  technology  in  detail  but  have  a  technical  appreciation  of  the 
need,  and  know  whether  the  work  adequately  addresses  that  need.  If  the  review  is  to 
address  both  technical  quality  and  relevance,  the  membership  of  the  panel  will  be  mixed. 
One  aspect  of  quality  review  is  assessment  of  the  laboratory  environment — the  suitability 
of  the  laboratory  equipment  and  buildings,  the  characteristics  of  the  staff,  support  groups, 
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such  as  the  machine  and  electronic  shops,  the  support  and  encouragement  given  by  the 
senior  managers  of  the  laboratory,  and  the  adequacy  of  the  budget.  To  judge  these 
aspects,  the  panel  will  necessarily  need  members  who  are  researchers  with  experience  at 
similar  laboratories. 

Following  this  general  discussion  of  peer  review,  Chapter  2  presents  the  many  ways  the 
Army’s  S&T  laboratories  conduct  reviews.  Chapter  3  then  discusses  several  approaches 
now  being  used  at  other  Federal  laboratories.  Chapter  4  discusses  the  findings  of  this 
investigation.  Chapter  5  provides  closing  comments  and  recommendations. 

The  paper  recommends  that  the  Army  require  peer  review  of  the  technical  quality  of  its 
laboratories  and  proposes  a  set  of  nonns  that  must  be  met.  The  principal  recommendation 
is  that  reviews  be  performed  by  independent  experts  who  visit  the  laboratory  for  two  or 
more  days,  looking  at  the  technical  projects  and  the  strength  of  the  technical  staff, 
equipment,  and  facilities.  The  recommendations  include  a  caveat  about  potential  conflicts 
of  interest  in  these  panels. 
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Chapter  2:  Peer  Review  at  Army  Laboratories 


To  discover  how  the  various  Army  laboratories  carry  out  technical  peer  review  of  their 
work,  interviews  were  conducted  with  leaders  of  each  of  the  four  segments  of  the  Army 
that  run  research  programs;  namely,  the  Army  Materiel  Command’s  Research, 
Development,  and  Engineering  Command  (RDECOM)  and  its  laboratories,  the  Medical 
Research  and  Materiel  Command  (MRMC),  the  Corps  of  Engineers  (COE),  and  the 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI).  For  RDECOM, 
discussions  were  held  with  seven  of  the  ten  technical  components — those  that  conduct 
bench-level  research  and  development.  Our  investigation  showed  that  the  Army  uses  a 
variety  of  approaches  to  evaluate  the  performance  of  its  research  and  development  work. 
The  following  sections  summarize  the  findings. 

The  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  (Research  and 
Technology) 

This  office  conducts  many  levels  and  kinds  of  reviews,  two  of  which,  done  by  its 
Director  of  Research  and  Laboratory  Management  (DRLM),  are  pertinent  here.  The  first 
is  the  Basic  Research  Review;  the  other  is  the  U.S.  Army  Research  and  Development 
Laboratory  of  the  Year  review.  Both  reviews  use  external  panels  of  experts. 

The  DRLM  appoints  a  set  of  independent  review  panels  that  review  the  Army’s  basic 
research  (referred  to  as  “6.1”),  including  in-house  laboratory  innovative  research  (ILIR). 
Rather  than  reviewing  individual  laboratories,  the  review  considers  broad  technical  areas 
that  may  include  the  work  of  more  than  one  laboratory.  The  DRLM  also  appoints  an 
external  Army  Science  Assessment  Group  that  reviews  the  findings  of  the  independent 
review  panels  and  makes  comments  and  recommendations  to  the  DRLM.  This  activity  is 
performed  every  3  years. 

In  the  Army  Laboratory  of  the  Year  competition,  each  laboratory  makes  a  submission 
covering  many  different  aspects  of  their  work.  One  of  these  is  a  description  of  what  they 
consider  to  be  the  single  most  significant  accomplishment  in  fundamental  research; 
another  is  a  description  of  what  they  consider  to  be  the  single  most  significant 
accomplishment  in  development  work.  The  DRLM  appoints  an  Army  Laboratory 
Assessment  Group  (ALAG),  comprising  external  experts  in  relevant  fields,  to  review  and 
grade  these  submissions.  The  ALAG  then  sends  a  member  to  each  of  twelve  laboratory 
installations  for  a  day-long  on-site  review.  Technical  peer  quality  review  of  the  research 
programs  is  only  a  small  part  of  this  program. 

The  Army  Materiel  Command’s  Research  Development  and  Engineering 
Command 

The  Research,  Development,  and  Engineering  Command  of  the  Army  Materiel 
Command  (AMC)  manages  the  AMC  laboratories,  which  consist  of  the  Army  Research 
Laboratory  (ARL)  and  its  Army  Research  Office  (ARO),  and  six  Research,  Development 
and  Engineering  Centers  (RDECs). 
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The  Army  Research  Laboratory — ARL  contracts  with  the  National  Research 
Council  (NRC)  of  the  National  Academies  to  assemble  an  ARL  Technical 
Assessment  Board  (TAB).  The  TAB  consists  of  six  panels,  each  composed  of  eight 
to  ten  individuals  having  high  repute  within  the  technical  community.  Each  panel 
reviews  one  of  ARL's  primary  technical  mission  areas.  The  Board,  with  its  panels, 
provides  an  appraisal  of  the  scientific  and  technical  efforts  of  ARL.  It  is  specifically 
enjoined  from  making  judgments  on  the  programmatic  structure  of  ARL’s  work, 
partly  because  it  is  not  equipped  to  perform  this  function,  and  partly  because  there 
are  numerous  other  channels  through  which  ARL  receives  such  programmatic 
guidance.  (Examples  include:  the  normal  chain  of  command  above  ARL,  the  Deputy 
Assistant  Secretary  of  the  Army  for  Research  and  Technology,  the  Director  of 
Defense  Research  and  Engineering,  the  Training  and  Doctrine  Command,  and  the 
entire  planning  and  budgeting  process  to  include  Congressional  committees.) 

Because  of  the  size  and  diversity  of  the  ARL  technical  program,  the  panels  review 
the  total  program  in  depth  over  a  3 -year  period.  There  is  a  formal  report  from  the 
TAB  every  2  years.  The  panels  produce  a  written  assessment  that  is  published  by  the 
NRC.  The  TAB  also  meets  with  the  ARL  director  to  provide  an  informal  report  of 
its  findings  before  the  publication  of  the  report,  and  to  receive  guidance  on  the 
director's  desires  for  special  areas  of  emphasis  for  the  subsequent  year's  review. 

The  AMC  Research,  Development  and  Engineering  Centers — All  of  the  RDECs  work 
closely  with  acquisition  program  executive  officers  and  program  managers,  and  draw  a 
considerable  fraction  of  their  support  from  these  sponsors.  RDECs  have  relatively  little 
basic  research  funding;  what  little  they  have  is  from  a  special  fund  called  Independent 
Laboratory  Innovative  Research  (ILIR).  Thus,  peer  review  will  emphasize  the  customer 
for  the  work,  hence,  relevance,  timeliness,  and  quality. 

None  of  the  six  AMC  centers  that  conduct  extensive  in-house  laboratory  research  conduct 
peer  review  of  the  kind  described  by  ARL,  but  some  are  considering  a  similar  approach. 
One  uses  a  contractor  to  put  together  a  panel  to  conduct  biannual  technical  reviews  for  its 
6.2  and  6.3  work  (it  has  very  little  6.1  money),  and  that  work  is  reviewed  in  any  case  by 
the  DRLM’s  Basic  Research  Review.  Another  RDEC  began  an  external  review  in  2007 
by  contracting  to  obtain  a  panel  of  outside  experts  to  review  one  major  technical  area  of 
the  RDEC  for  a  given  year;  subsequently  another  panel  will  be  set  up  to  review  another 
segment  the  following  year. 

Another  RDEC  is  considering  the  introduction  of  formal  technical  reviews  by  external 
experts,  but  has  not  yet  selected  an  approach.  The  remaining  three  RDECs  are  not  using 
external  technical  peer  review.  It  should  be  noted  that  all  the  AMC  RDECs  conduct 
extensive  internal  program  reviews  for  RDECOM  and  frequent  reviews  with  their 
customers. 
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Details  of  the  interviews  at  the  RDECs  follow: 


Aviation  and  Missile  Research,  Development,  and  Engineering  Center — AMRDEC 
relies  on  two  processes  to  review  its  programs.  The  technical  quality  is  briefly  looked  at 
in  the  Army's  Laboratory  of  the  Year  program.  The  panel  of  judges  is  composed  of  senior 
technologists  external  to  the  Anny.  They  review  AMRDEC ’s  nomination  package,  and  a 
panel  member  visits  AMRDEC  for  a  day.  The  AMRDEC  program  is  also  reviewed  by  a 
panel  consisting  primarily  of  retired  general  officers,  with  some  representatives  from 
senior  S&T  leadership.  Panel  members  are  chosen  to  have  a  diverse  set  of  backgrounds  to 
insure  balance  in  their  advice.  This  panel  serves  as  a  liaison  for  AMRDEC  to  the  rest  of 
the  Army. 

Armaments  Research,  Development,  and  Engineering  Center — ARDEC  uses  a 
contractor  to  put  together  biannual  technical  peer  reviews  of  6.2  and  6.3  work.  (The  only 
basic  research  they  have  is  the  in-house  laboratory  innovative  research  (ILIR)  funding). 
The  panels  are  made  up  of  technical  specialists  from  across  DOD,  including  some  retirees 
from  the  uniformed  Army.  The  panel  members  are  paid  for  expenses  and  time,  although 
some  members  will  not  accept  remuneration.  The  panel  receives  briefings  for  2-2  !4  days 
at  the  laboratory  to  assess  quality  and  content,  and  spend  additional  time  writing  a  report. 
The  lab  is  reviewed  this  way  every  2  years.  The  review  uses  a  checklist  designed  to  look 
at  both  individual  program  elements  and  the  lab  as  a  whole.  The  ARDEC  presentations 
compare  the  work  to  other  labs,  such  as  Los  Alamos  or  Livermore  National  Laboratories. 

Natick  Soldier  Research,  Development,  and  Engineering  Center — NSRDEC  brings  in 
university  academics  to  look  at  6.1.  NSRDEC  has  relied  on  the  Army  Science  and 
Technology  Working  Group  (ASTWG)  process  for  additional  review  of  outcomes  and 
relevance,  but  does  not  carry  out  technical  reviews.  These  reviews  do  not  evaluate  the 
quality  of  people,  equipment  etc.  The  Army  Laboratory  of  the  Year  competition  review 
sends  an  outside  expert  to  NSRDEC  for  one  day.  This  person  obtains  an  overview  and  a 
tour  of  the  laboratories.  A  technical  peer  review  is  not  conducted. 

The  NSRDEC  management  is  now  ready  to  look  at  comprehensive  peer  review  for  6.2 
(and  beyond)  work.  They  want  a  methodology  for  looking  at  both  quality  and  relevance. 
They  want  to  evaluate  technical  quality,  relevance,  quality  of  individual  skills,  and  the 
skill  mix.  They  are  also  interested  in  benchmarking  with  other  laboratories.  They  have 
had  a  prior  process  (described  in  a  2006  RDECOM  report5),  but  discontinued  that 
approach  after  judging  it  insufficient  for  peer  review — it  was  more  of  an  information 
sharing  effort. 

Edgewood  Chemical  Biological  Center — The  ECBC  technical  director  first  discussed 
the  Army-wide  laboratory  of  the  year  program,  since  the  ALAG  had  just  visited.  The 
ALAG  visitors  impressed  them,  and  ECBC  in  particular  liked  the  on-site  feedback  they 
received  from  the  visitors.  The  visit  was  for  one  day  only;  a  second  day  would  have 
provided  time  for  more  detailed  presentations  and  visits  to  staff  in  their  laboratories. 


5  RDECOM  Committee,  Chair  Dr.  J  Mait,  “RDECOM  Initiative  on  Peer  Review  and  Benchmarking” ,  Final 
Report,  September  2006.  Available  from  Dr.  J.  Mait,  Army  Research  Laboratory,  Adelphi,  MD. 
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ECBC  believes  the  visit  would  have  been  more  useful  had  it  not  been  tied  to  a 
competition  and  ultimately  an  award  to  one  laboratory.  (The  award  process  was  felt  to 
have  limited  the  feedback  to  ECBC  at  the  time  of  the  visit,  because  some  of  the  criticism 
would  be  used  subsequently  in  the  award  decision.) 

ECBC  has  used  an  eight-member  focal  group  and  facilitator  to  look  at  some  of  their 
programs.  Members  include  retired  generals  and  some  technical  people.  They  have 
looked  at  organizational  issues  such  as  the  state  of  the  facilities,  and  the  technical  merit  of 
the  work.  They  have  provided  a  written  report.  ECBC  has  used  IDA,  RAND  and  the 
National  Academies  for  studies  focused  on  narrow  areas  of  technology;  for  example, 
toxicity  work  at  ECBC.  The  studies  include  visits  to  the  labs.  These  are  not  intended  to  be 
comprehensive  and  should  not  be  construed  as  related  to  an  overall  peer  review  process. 

Communications  and  Electronics  Research,  Development,  and  Engineering  Center — 
CERDEC  does  not  have  a  technical  peer  review  process  in  the  sense  used  in  this  paper. 
CERDEC  relies  solely  on  the  laboratory  of  the  year  ALAG  visits  for  external  assessment. 
However,  a  former  technical  director  had  earlier  invited  groups  of  “graybeards”  to  the 
laboratory  periodically  to  review  the  technical  work.  This  process  was  unstructured, 
informal,  and  eventually  discontinued. 

In  regard  to  the  process  for  the  Army  Laboratory  of  the  Year,  CERDEC  offered  some 
criticisms.  CERDEC  claims  not  to  receive  formal  feedback  for  two  months  or  more  after 
the  visit.  The  funds  allocated  by  DRLM  for  the  ALAG  visits  evidently  will  not  allow  for 
visits  to  both  Ft  Monmouth  and  Ft  Belvoir,  so  the  ALAG  is  able  to  visit  only  part  of  the 
RDEC  each  year. 

This  year,  the  ALAG  visitor  had  a  security  clearance  and  was  able  to  go  into  the 
classified  facility  and  receive  detailed  briefings  not  available  to  previous  ALAG  visitors. 
CERDEC  believes  the  ALAG  visits  are  too  short  and  too  limited  as  to  the  amount  of 
information  that  can  be  provided.  They  also  find  that  statements  made  by  the  visitors 
during  the  visits  do  not  always  agree  with  the  formal  feedback  received  much  later.  They 
believe,  with  ECBC  and  NSRDC,  that  the  ALAG  process  would  be  better  without 
limiting  the  competition  for  an  award  to  just  one  laboratory.  They  believe  that  a  longer 
visit,  say  two  or  more  days,  using  a  panel  of  experts  would  improve  the  process  by 
allowing  more  in-depth  discussion  and  more  visits  to  laboratories. 

Tank  and  Automotive  Research,  Development,  and  Engineering  Center — TARDEC 
uses  senior  technical  personnel,  including  two  STs  (senior  technical  positions  without 
administrative  duties  and  equivalent  to  positions  in  the  Senior  Executive  Service)  for 
internal  technical  reviews.  On  a  quarterly  basis  they  do  full  program  reviews.  They  also 
do  an  internal  technical  peer  review  before  starting  new  programs.  The  internal  reviews 
also  include  work  done  by  Small  Business  Innovation  Research  (SBIR)  program  grants 
and  all  Congressional  add-ons. 

TARDEC  has  begun  outside  peer  review  by  contracting  with  the  Institute  for  Defense 
Analysis  (IDA)  to  select  a  panel  of  outside  experts.  The  first  review  was  in  2007  and  was 
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on  survivability  technology.  They  plan  to  continue  these  reviews;  next  up  are  power  and 
energy  technologies.  The  expert  panels  have  five  to  ten  members  who  meet  in  3 -day 
workshops.  This  process  reviews  just  one  topical  area  each  year;  TARDEC  has  found  this 
to  be  a  very  effective  way  to  evaluate  technical  quality. 

The  technical  director  believes  that  the  laboratory  of  the  year  program  does  not  meet  the 
criteria  of  a  peer  review  process  for  technical  work.  The  technical  director  expressed 
interest  in  the  National  Research  Council  review  program. 

Medical  Research  and  Materiel  Command — All  of  the  MRMC’s  R&D  is  externally 
reviewed  under  a  contract  managed  by  the  American  Institute  of  Biological  Sciences 
(AIBS),  formerly  a  part  of  the  National  Academies  and  now  independent.  Review  is  done 
using  panels  consisting  of  distinguished  external  experts  who  possess  the  appropriate 
subject  matter  expertise.  These  reviews  are  for  research  proposals  made  to  MRMC. 
Intramural  researchers  must  also  make  annual  proposals  for  their  work;  these  too  are 
reviewed  by  AIBS  panels.  The  research  program  at  MRMC  is  divided  into  four  program 
elements,  each  of  which  is  overseen  by  a  program  manager  at  headquarters.  Every  3-5 
years,  each  program  element  brings  in  a  review  panel  to  look  at  both  quality  and  program 
content.  Most  reviews  are  conducted  by  AIBS;  on  occasion  reviews  are  obtained  through 
the  National  Academies’  Institute  of  Medicine  or  the  National  Research  Council.  The 
quality  of  facilities  and  equipment  are  only  sporadically  reviewed;  comments  on  facilities 
and  equipment  are  sometimes  offered  by  panel  members  during  site  visits. 

Engineer  Research  and  Development  Center — The  Army  Corps  of  Engineers’  ERDC 
consists  of  seven  laboratories  at  four  sites.  Each  laboratory  is  reviewed  every  2  years  by 
external  panels  of  about  ten  experts.  The  ERDC  director  manages  the  selection  of  experts 
and  the  meeting  agendas.  The  panels  review  quality  of  work,  staff,  equipment,  and 
facilities.  The  panels  also  look  at  program  relevance.  The  panels  provide  a  qualitative 
ranking  comparing  the  ERDEC  programs  to  the  best  peer  laboratories  around  the  world. 
The  reviews  usually  last  3  days.  Written  reports  are  submitted,  usually  within  a  month 
after  the  meetings.  The  panels  review  technical  programs  in  detail  and  visit  the  laboratory 
staff  at  the  bench  level.  Panel  members  are  nominated  by  each  of  the  seven  laboratory 
directors  and  approved  by  the  ERDC  director.  The  members  are  paid  for  their  time.  There 
is  some  dual  membership  across  the  panels.  Members  are  from  industry  and  retired 
military  and  include  several  members  of  the  National  Academies.  Some  are  eminent  in 
their  respective  fields;  some  are  generalists.  There  is  a  standard  protocol  for  the  labs  to 
follow  in  conducting  their  biannual  reviews. 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences — ARI  has  124 
authorized  positions  and  a  budget  of  $32M.  About  half  of  the  budget  is  spent  outside  the 
Institute  on  contracts  or  grants.  Last  year  the  6.1  budget  was  doubled  to  $6.0  million, 
thereby  allowing  ARI  to  broaden  and  deepen  its  research.  ARI  operates  its  own  office  to 
solicit,  evaluate,  and  allocate  research  grants.  The  ARI  has  no  laboratories  in  the  usual 
sense.  Instead  it  has  staff  distributed  across  many  different  Army  Training  and  Doctrine 
Command  (TRADOC)  centers  and  schools. 
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ARI  sometimes  has  joined  with  ARL’s  Human  Research  and  Engineering  Directorate  for 
research  reviews  of  focused  areas.  ARI  believes  that  quality  can  be  measured  by  success 
in  transition  and  adoption,  by  the  Army,  of  their  recommendations.  This  is  not,  however, 
a  technical  peer  review. 

ARI  has  the  NRC  Board  on  Behavioral,  Cognitive,  and  Sensory  Sciences  study  possible 
new  areas  for  work  every  3-4  years.  For  external  scientific  reviews,  ARI  uses  advisory 
groups  that  include  both  internal  and  external  panel  members.  There  is  a  formal  panel  on 
leader  development,  a  small  TRADOC  panel  on  training,  and  a  mixed  military/science 
panel  on  personnel.  The  ARI  staff  who  work  in  a  given  area  of  technology  attend  these 
reviews  and  receive  instant  feedback.  The  advice  is  usually  verbal  and  not  in  written 
reports. 


11 


Chapter  3:  Peer  Review  at  Some  Other  Government 
Laboratories 


At  this  point  it  is  useful  to  review  approaches  to  peer  review  used  at  other  DOD 
laboratories  and  at  laboratories  at  other  Federal  agencies,  either  owned  and  operated  by 
the  government  or  contracted  by  the  government  to  private  entities.  It  should  be  noted 
that,  as  was  the  case  with  the  Army  laboratories,  peer  review  is  done  in  several  different 
ways  at  these  other  laboratories.  One  type  of  review  is  that  done  by  the  National 
Academies’  National  Research  Council  (NRC).  The  NRC  has  study  boards  for  all  three 
Services.  The  Boards  may  perform  in-depth  technical  reviews.  The  Defense  Science 
Board  (DSB)  has  reviewed  the  Defense  laboratories,  but  not  in  terms  of  specific  technical 
work.  Rather,  the  DSB  has  looked  at  DOD’s  policies  for  the  laboratories — a  higher  level 
of  review.  One  aspect  of  external  review  is  that  the  Federal  Advisory  Committee  Act 
(FACA)  may  govern  the  operation.  This  means  that  meetings  must  be  public,  members  of 
the  public  may  make  presentations  to  the  committee,  and  members  are  appointed  only 
with  concurrence  of  the  President’s  agents  (typically  the  General  Services  Administration 
or  agency  heads).  These  requirements  would  have  infringed  on  the  traditional  operations 
of  the  National  Academies.  After  consideration,  the  FACA  was  amended  to  exempt  the 
Academies  from  many  of  the  requirements. 

The  Services  have  traditionally  convened  their  own  advisory  committees  to  look  at 
technology  issues,  e.g.,  the  Army  Science  Board,  the  Air  Force  Science  Advisory  Board, 
the  Naval  Studies  Board,  and  the  Naval  Research  Advisory  Committee.  These  groups 
differ  in  their  operations.  Some  are  involved  in  detailed  technical  studies,  others  restrict 
themselves  to  policy  studies.  All  of  them  come  under  the  FACA.  On  occasion,  ad  hoc 
advisory  committees  may  be  set  up,  often  by  direction  of  the  U.S.  Congress.  Two 
examples  are  the  Federal  advisory  commission  established  to  review  the  Services’  plans 
for  realigning  their  laboratories  during  the  base  realignment  and  closure  activity  for 
1991, 6  and  a  committee  chartered  by  the  Congress  to  study  the  future  relevance  of  DOD 
laboratories.7  A  series  of  reports  on  visits  by  the  latter  committee  to  several  DOD 
laboratories  was  issued  in  2002.  These  reports  were  the  result  of  one-time  reviews,  after 
which  the  committee  in  question  ceased  to  exist.  Congressional  committees,  the 
Government  Accountability  Office,  and  even  the  Library  of  Congress  have  also  done 
policy  studies  of  the  laboratories. 


6  Federal  Advisory  Commission  on  Consolidation  and  Conversion  of  Defense  Research  and  Development 
Laboratories,  Report  to  the  Secretary’  of  Defense  (Washington,  DC:  Department  of  Defense,  September 
1991). 

7  National  Defense  Authorization  Act  of  FY  2000,  S-1059,  106th  Congress,  Section  913. 

s  National  Defense  University,  Section  913  Report  #1,  #2,  #3,  (Washington,  DC:  Center  for  Technology 
and  National  Security  Policy,  2002).  Copies  of  this  report  are  available  from  the  Office  of  the  Director, 
Center  for  Technology  and  National  Security  Policy,  National  Defense  University. 


12 


Assessment  of  the  Laboratories  at  the  National  Institute  of  Standards  and 
Technology 

The  programs  at  the  National  Institute  of  Standards  and  Technology  (NIST),  formerly  the 
National  Bureau  of  Standards  (NBS),  have  been  assessed  by  the  National  Academies’ 
National  Research  Council  (NRC)  since  the  1950s.  This  was  among  the  first  regularly 
recurring  quality  reviews  of  a  Federal  laboratory  by  the  NRC,  and  it  continues  to  this  day. 
(Recently  the  review  has  been  emulated  at  the  Army  Research  Laboratory.  See  discussion 
of  ARL  in  chapter  2.) 

Peer  review  by  the  NRC  began  at  NBS  after  a  prolonged  controversy  over  NBS’ 
evaluation  of  a  commercial  product.  After  these  issues  were  settled  in  NBS’  favor,  the 
Secretary  of  Commerce  asked  the  National  Academy  of  Science  (now  the  National 
Academies)  the  parent  of  the  NRC,  to  study  various  aspects  of  NBS  activities.  One  result 
was  the  establishment  of  a  separate  NRC  board  to  assess  on  a  regular  basis  the  quality  of 
NBS  work  and  the  factors  affecting  that  work.  A  set  of  evaluation  panels  was  created 
under  the  NRC  Board  on  Assessment  for  the  National  Bureau  of  Standards  (now  the 
NRC  Committee  on  NIST  Technical  Programs).  These  panels  met  once  a  year  (more 
recently  every  other  year),  generally  for  2  days,  to  review  technical  work,  visit  the 
laboratories,  view  buildings  and  equipment,  and  talk  with  staff.  They  provided  a  written 
report  vetted  by  the  NRC  and  published  to  the  general  public.  Typically  there  were  (and 
are)  separate  panels  for  each  major  operating  unit  at  NBS. 

The  panels  occasionally  look  at  cross-cutting  research  that  spans  more  than  one  division. 
On  occasion,  the  NIST  director  requests  attention  to  certain  areas,  e.g.,  quality  of  support 
services,  state  of  buildings  and  working  laboratory  spaces,  and  status — age,  morale,  and 
so  on — of  technical  staff. 

From  the  outset,  the  panels  were  made  up  of  experts  drawn  mostly  from  outside  the 
Federal  government.  Members  may  be  nominated  by  anyone,  but  the  selections  are 
controlled  by  the  NRC.  Panel  members  are  not  paid,  except  for  travel  expenses.  (A  firm 
NRC  policy  holds  that  none  of  its  committee  members  should  receive  remuneration.)  The 
assessment  work  is  scheduled  and  managed  by  NRC  staff  dedicated  to  the  purpose. 
Reports  are  published  by  the  National  Academies  Press  and  are  not  restricted. 

The  individual  panels  are  overseen  by  the  NRC  Committee  on  NIST  Technical  Programs. 
The  chairman  of  the  committee  often  has  joined  in  discussions  with  the  NIST  statutory 
visiting  committee  about  NIST,  both  in  presentations  to  the  visiting  committee  and  in 
presenting  findings  to  the  Secretary  of  Commerce  and  the  NIST  authorization  committees 
in  Congress. 

Other  Approaches 

Interviews  with  senior  officials  at  a  number  of  Federal  laboratories  have  revealed  several 
different  models  for  peer  review.  Some  are  fonnal  review  panels  that  come  under  the 
FACA;  others  are  fonnal  but  managed  internally  by  the  Service  or  the  laboratory.  An 
example  of  one  managed  by  a  Service  is  the  review  by  the  Air  Force  Scientific  Advisory 
Board  (SAB). 
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The  Air  Force  Research  Laboratory — The  SAB,  a  formal  Federal  Advisory 
Committee,  was  set  up  shortly  after  World  War  II  by  General  Henry  (“Hap”)  Arnold  and 
headed  by  Dr.  Theodore  von  Karman.  At  first  known  as  the  Scientific  Advisory  Group,  it 
prepared  a  landmark  series  of  reports  known  collectively  as  “Toward  New  Horizons.”9 
This  work  focused  on  the  then  state-of-the-art  and  outlined  detailed  plans  for  research  in 
the  sciences  and  technologies  deemed  critical  to  the  future  of  air  power.  The  group 
reported  directly  to  Gen.  Arnold.  Today,  as  the  SAB,  it  continues  to  report  to  the  Air 
Force  Chief  of  Staff  and  the  Secretary  of  the  Air  Force.  Among  other  duties,  the  SAB 
reviews  in  depth  the  scientific  work  of  the  Air  Force  Research  Laboratory  (AFRL)  on  a 
2-year  cycle.  The  review  covers  both  the  6. 1-6.3  work  but  also  work  sponsored  by 
downstream  users,  such  as  program  managers  or  executive  officers.  One  panel  has  been 
created  for  each  AFRL  directorate.  Panel  members  are  selected  by  the  SAB;  all  are 
external  to  the  Air  Force.  Most  are  members  of  the  SAB,  but  when  necessary  the  SAB 
brings  in  consultants.  At  least  one  member  of  each  panel  must  be  a  member  of  the 
National  Academies. 

The  SAB  conducts  reviews  of  each  directorate,  devoting  a  full  week  to  each.  The  panels 
provide  an  exit  briefing  and  a  formal  report.  The  reports  are  sometimes  made  available  to 
the  public,  but  some  may  be  restricted  (FOUO,  for  official  use  only).  The  panels  look  at 
four  factors  for  the  programs:  technical,  relevance  for  the  near  tenn  (5  yrs),  future 
impacts,  and  resources.  When  evaluating  the  6.1  programs,  the  SAB  does  not  look  at 
near-term  impacts.  For  the  technical  work  (factor  1),  SAB  evaluates  technical  innovation, 
technical  rigor,  productivity,  and  collaboration. 

The  Naval  Research  Laboratory — Two  external  groups  have  been  established  to  look 
at  various  technical  topics  for  the  Navy:  the  NRC’s  Naval  Studies  Board,  and  the  Naval 
Research  Advisory  Committee,  created  by  the  Chief  of  Naval  Operations.  Both  groups 
consider  the  impact  of  technical  developments  on  the  future  of  naval  forces.  For  detailed 
technical  review  of  its  technical  base  research  programs,  the  Naval  Research  Laboratory 
(NRL)  establishes  and  manages  its  own  peer  review.  NRL  has  seven  focus  areas: 
materials  and  chemistry,  electronics,  battlespace  environment,  underseas  warfare, 
electromagnetic  warfare,  space  research/space  technology,  and  information  technology. 

Technical  review  of  the  technical  base  covers  about  one-third  of  the  total  research 
program  each  year.  NRL  selects  members  for  the  external  review  panels  for  each  focus 
area.  A  panel  typically  has  four  to  six  members  drawn  from  academia  and  elsewhere, 
along  with  at  least  one  member  of  the  National  Academies.  The  NRL  asserts  that  these 
members  are  unbiased.  The  panels  meet  for  from  two  to  four  days  with  time  for 
immersion  in  the  labs  and  staff.  The  panels  give  exit  briefings  to  the  NRL  management. 
Subsequently,  they  submit  a  formal  written  report  of  about  10-15  pages,  and  NRL 
responds  to  this  in  writing. 


9  Michael  H.  Gorn  (editor),  Prophecy’  Fulfilled,  ‘Toward  New  Horizons  ’  and  its  Legacy  (Washington,  DC: 
Air  Force  Historical  Studies  Office,  1994).  Available  at  <http://www.airforcehistory.hq.af.mil/ 
Publications/authorindex.htm/> 
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The  panels  evaluate  the  programs  for  scientific  merit:  they  examine  the  research 
approach,  the  credentials  of  the  staff,  the  project’s  relevance,  equipment,  and  costs. 
Details  of  these  differ  for  6.1  and  6.2.  For  6.1,  the  panels  look  for  work  that  seeks  to 
expand  the  frontiers  of  known  science;  for  6.2  the  panels  look  for  whether  NRL  is 
investigating  and  developing  recent  advances  in  science  and  technology.  They  have  seven 
categories  of  evaluation  containing  metrics.  For  the  customer- funded  work,  the  criteria 
used  are  those  of  the  customers. 

The  National  Institutes  of  Health — The  NIH  operates  peer  review  by  nineteen  formal 
advisory  committees,  termed  boards  of  scientific  counselors,  one  for  each  of  the 
institutes.  Their  duties  are  described  as  follows:  “Boards  of  Scientific  Counselors  serve  a 
dual  function  in  providing  expert  scientific  advice  to  Scientific  Directors  regarding 
particular  employees  and  projects,  and  providing  the  NIH  as  a  whole  with  an  assessment 
of  the  overall  quality  of  its  intramural  efforts.”10 

There  is  a  Committee  Management  Office  at  NIH  to  track  these  and  many  other  NIH 
advisory  committees.  The  office  stays  in  contact  with  the  FACA  Office  of  the  General 
Services  Administration.  The  NIH  process  is  the  most  formal  type  of  quality  review 
discovered  in  interviews  for  this  study. 

Three  National  Laboratories  of  the  Department  of  Energy — Los  Alamos  National 
Laboratory  (LANL),  Lawrence  Livermore  National  Laboratory  (LLNL),  and  Sandia 
National  Laboratory  (SNL)  are  government-owned,  contractor-operated  facilities  under 
the  Department  of  Energy  (DOE).  Interviews  at  these  three  laboratories  revealed  very 
similar  processes  for  peer  review.  The  Department  of  Energy  National  Nuclear  Security 
Administration  (NNSA)  contracts  for  LANL  and  LLNL  explicitly  call  for  regular  peer 
review;  Sandia’s  contract  contains  similar  language.  LANL  and  LLNL  are  operated  by  a 
consortium  of  the  University  of  California  and  the  Bechtel  Corporation,  with  additional 
partners  from  the  Babcock  and  Wilcox  Company  and  the  Washington  International 
Group.  Two  limited  liability  corporations  have  been  created  to  manage  the  contracts:  the 
Los  Alamos  National  Security  LLC,  and  the  Lawrence  Livermore  National  Security 
LLC.  There  is  a  joint  Board  of  Governors  for  the  two  LLCs.  The  Board  has  a 
Subcommittee  on  Science  and  Technology  (the  S&T  Committee)  that  oversees  and 
controls  peer  reviews  of  S&T  at  both  laboratories.  The  S&T  Committee  makes  the 
appointments  to  the  peer  review  panels  upon  nomination  by  the  laboratories.  The  panels 
are  independent  and  balanced.  Each  does  both  review  and  critique.  The  panels  have  eight 
to  ten  members  drawn  from  academia  (about  half),  industry,  and  other  labs;  there  are 
some  University  of  California  faculty,  and  some  from  other  National  Laboratories.  The 
operation  of  the  panels  does  not  come  under  the  FACA. 

Los  Alamos  National  Laboratory  conducts  three  kinds  of  review.  The  first  review  is 
strictly  concerned  with  quality  of  scientific  activity  and  is  focused  on  capability  areas 
rather  than  on  scientific  disciplines.  Typically,  these  are  crosscuts  from  the  discipline 


10  Directory  of  NIH  Federal  Advisory  Committees  and  HHS  Advisory  Committees  Managed  by  NIH  Staff, 
(Washington,  D.C.:  National  Institutes  of  Health,  January  2008),  6.  Available  at 
<wwwl. od.nih.gov/cmo/committees/directoryofnihadvisorycommittees>. 
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areas,  such  as  weapons  science  and  information  science.  They  cover  eight  capabilities  per 
year;  each  capability  area  is  reviewed  every  3  years.  The  review  panels  meet  for  3-4 
days.  (In  earlier  times  these  reviews  were  run  strictly  in-house  and  covered  only  scientific 
disciplines,  not  capabilities.)  The  panels  also  look  at  the  adequacy  of  the  lab 
infrastructure,  the  morale  of  the  staff,  and  the  research  environment.  The  other  reviews 
are  on  weapons  design  and  customer  programs.  The  design  reviews  are  done  internally  by 
DOE  weapons  design  teams;  the  customer  reviews  look  at  quality,  relevance,  and 
performance  against  the  mission.  Customer  reviews  are  set  up  by  the  laboratory  subject  to 
approval  by  the  Board  of  Governors’  S&T  Committee. 

Lawrence  Livermore  National  Laboratory  follows  a  similar  assessment  process.  At 
LLNL,  there  are  four  principal  disciplines  and  three  major  program  areas.  All  are 
reviewed  by  peer  panels.  In  addition,  they  conduct  cross-cutting  reviews  of  portfolios  (for 
example,  the  National  Ignition  Facility)  involving  more  than  one  of  the  seven  areas.  The 
panels  consist  of  10-15  members  drawn  from  academia,  industry,  and  other  laboratories. 
Members  are  selected  by  the  LLNL  directorates  and  vetted  by  the  Board  of  Governors’ 
S&T  Committee.  Selection  factors  include  diversity  on  the  panels  and  turnover  of 
membership.  The  panels  usually  have  a  member  from  LANL  and  someone  from  NNSA. 
Panel  reports  contain  both  critique  and  advice.  Verbal  exit  briefings  are  given  to  the  head 
of  the  unit  under  review,  as  well  as  to  senior  managers  and  the  LLNL  director.  Reports  of 
the  panels  are  circulated  within  DOE  but  are  restricted. 

Sandia  National  Laboratory  is  operated  under  a  contract  between  NNSA  and  Sandia 
Corporation,  a  wholly  owned  subsidiary  of  the  Lockheed  Martin  Corporation.  There  are 
six  Lockheed  employees  on  the  Sandia  Corporation  Board  of  Directors.  The 
Corporation’s  S&T  Committee  is  chaired  by  the  Chief  Technical  Officer  of  Lockheed. 
About  half  of  the  $2  billion  plus  budget  at  Sandia  National  Laboratory  is  S&T. 

Peer  review  at  Sandia  is  divided  between  S&T  and  nuclear  weapons  work.  For  S&T,  the 
reviews  are  conducted  under  contract  to  the  University  of  Texas  (UT),  which  has  two 
positions  on  the  Board  of  Sandia  Corporation.  Under  supervision  by  the  Sandia 
Corporation  Board  of  Directors  Subcommittee  on  S&T,  UT  selects  and  convenes  review 
panels.  Both  Sandia  management  and  Sandia  Corporation  have  input  into  the  final 
selections. 

Each  panel  has  6-12  members  from  various  disciplines,  including  physical  sciences, 
computation,  electronics,  and  materials.  UT  draws  some  panel  members  from  its  faculty; 
other  members  are  drawn  from  wherever  they  can  find  qualified  individuals.  There  is  an 
external  panel  for  each  competence.  Panels  meet  for  multi-day  sessions  and  cover 
individual  projects/programs.  The  panels  meet  with  staff  members  to  assess  morale  and 
the  research  environment.  They  also  may  meet  with  groups  of  principal  investigators  at 
the  program  level,  as  well  as  with  individual  project  leaders.  The  review  results  are 
reported  to  laboratory  management  and  the  Board  of  Directors’  S&T  Committee. 
Principal  Investigators  receive  the  reports  and  must  respond  to  panel  critiques. 
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Nuclear  weapons-related  peer  review  at  Sandia  includes  three  kinds  of  internal  reviews: 
design,  management,  and  internal  peer  review  (members  from  entities  other  than  the 
design  team  under  review).  Reviews  are  assisted  by  a  full-time  office  of  assessment  that 
reports  directly  to  the  laboratory  director.  The  assessment  staff  is  internal  but  separate 
from  the  program  areas.  There  is  also  a  standing  panel  for  external  independent  review  of 
these  Sandia  assessments.  These  assessments  are  a  critical  part  of  the  regular  certification 
of  America’s  nuclear  weapons  stockpile. 

Peer  review  at  grant-making  organizations 

Entities  that  award  grants — for  example,  the  National  Science  Foundation  (NSF),  the 
Army  Research  Office  (ARO),  the  Office  of  Naval  Research  (ONR),  and  the  Air  Force 
Office  of  Scientific  Research  (AFOSR) — do  not  operate  laboratories  and,  therefore, 
conduct  quality  reviews  in  a  different  manner.  NSF  is  overseen  by  the  National  Science 
Board  that  reports  to  the  Congress.  Each  NSF  directorate  is  monitored  by  a  fonnal 
advisory  committee  that  meets  regularly  to  review  performance.  Periodically,  members  of 
the  committee  review  the  grant  folders  to  ensure  that  procedures  have  been  followed. 
Grant  proposals  are  sent  out  to  experts  for  evaluation;  subsequently  the  folders  are 
evaluated  by  NSF  staff  before  the  decision  to  award  or  not  to  award  is  made.  For  work  in 
progress,  grantees  are  visited  on  site  by  NSF  program  managers.  Quality  is  judged  by 
these  reviews,  regular  reports,  and  copies  of  publications.  The  ultimate  test  of  how  well  a 
grantee  is  doing  is  the  renewal  or  termination  of  the  grant.  This  is  true  for  all  the  granting 
agencies. 

ARO  conducts  two  kinds  of  peer  review  concerning  single  investigator  (SI)  proposals  for 
new  work.  One  review  evaluates  technical  merit.  Typically  the  proposal  is  sent  to 
external  reviewers,  mostly  university  faculty.  The  other  review  focuses  on  military 
relevance  and  is  done  by  Anny  and  DOD  scientists  and  engineers.  The  SI  grants  are 
typically  for  3  years.  Two  or  more  site  visits  are  usually  conducted  during  this  time. 
Program  managers  will  audit  grantees  presentations  at  scientific  meetings  and  gauge 
audience  reaction.  ARO  receives  formal  annual  reports  and  copies  of  all  publications  by 
its  grantees. 

The  ARO  divisions  are  evaluated  biennially  by  external  boards  of  visitors,  one  for  each 
division.  The  boards  look  at  the  overall  portfolios,  evaluating  the  strategic  direction  of  the 
divisions  and  looking  out  for  overlap  with  other  programs  in  the  DOD  or  elsewhere. 
AFOSR,  a  directorate  within  AFRF,  manages  the  6.1  funds  for  the  Air  Force.  Funding 
executed  by  AFRF  internal  research  directorates  is  evaluated  by  the  SAB  during  the 
biennial  reviews  of  those  directorates.  The  AFOSR  as  a  whole  is  reviewed  by  the  SAB 
every  2  years. 

Office  of  the  Secretary  of  Defense 

The  Director  of  Defense  Research  and  Engineering  for  a  number  of  years  conducted 
Technology  Area  Reviews  and  Assessments  (TARA)  reviews  that  covered  DOD  basic 
research  programs.  Representatives  of  the  Service  laboratories  and  operating  commands 
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participated  in  these  reviews,  along  with  outside  experts.  TARA  reviews  are  no  longer 
conducted. 

Clearly  there  is  no  single,  accepted  best  way  to  conduct  peer  review  of  Federal  S&T 
laboratories.  Chapter  4  summarizes  the  results  of  the  preceding  investigation  into  the  peer 
review  process.  Recommendations  are  presented  in  chapter  5. 
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Chapter  4:  A  Summary  of  Peer  Review  Practices 


There  are  almost  as  many  approaches  to  peer  review  as  there  are  laboratories  contacted 
for  this  study.  Laboratory  practices  range  from  practically  no  review  to  formal  reviews 
conducted  under  the  Federal  Advisory  Committee  Act.  The  following  discussion  presents 
the  range  of  practices  uncovered  in  interviews  and  literature  study.  This  is  followed  by 
consideration  of  the  pros  and  cons  of  the  several  approaches.  Chapter  5  offers  a  set  of 
performance  guidelines  for  the  Army. 

Most  of  the  laboratories  studied  have  some  kind  of  peer  review.  At  one  end  of  the 
spectrum  stands  NIH,  which  has  an  external  advisory  committee  for  each  of  its  institutes. 
These  committees  operate  in  the  public  eye  under  the  FACA.  NIH  review  processes 
operate  under  a  very  formal  arrangement.  AFRL  has  a  formal  arrangement  with  the  Air 
Force  Science  Advisory  Board  (SAB),  operated  from  the  offices  of  the  Chief  of  Staff  and 
the  Secretary  of  the  Air  Force.  Thus,  the  SAB  is  once  removed  from  the  laboratory.  The 
SAB  also  takes  its  authority  from  the  FACA.  Another  formal  process  is  the  use  of 
dedicated  boards  and  committees  of  the  NRC,  under  contract  with  the  laboratories. 

Several  laboratories  manage  their  own  review  processes,  some  of  which  are  subject  to 
oversight  by  a  higher  level  of  management.  The  DOE  weapons  laboratories  are  reviewed 
by  expert  panels,  whose  nominations  from  laboratory  management  are  subject  to 
approval  by  the  contractor  that  operates  the  laboratory.  Some  of  the  Army  laboratories 
contract  out  at  least  a  portion  of  the  process.  Laboratory  directors  generally  value  outside 
review,  but  emphasize  that  the  reviewers  must  be  experts  in  the  specific  sciences  and 
technologies  being  studied  at  their  laboratories.  In  cases  where  the  technologies  are 
highly  specialized,  there  may  be  only  a  small  external  community  from  which  to  draw 
reviewers.  Furthermore,  these  may  be  somehow  involved  in  the  laboratory’s  research, 
making  conflict  of  interest  a  problem.  (NSF  and  the  NRC  address  this  issue  not  by 
disqualifying  an  otherwise  desirable  expert,  but  rather  by  making  the  potential  conflict 
known  and  balancing  the  various  interests  that  may  exist  on  their  panels.) 

Some  experts  are  drawn  from  universities  or  industry,  while  some  come  from  within  the 
Army  or  other  government  laboratories.  Several  Army  laboratories  use  exclusively 
external  review  panels.  Most  of  the  others  have  some  external  and  some  internal 
members.  No  evidence  was  found  of  unresolved  conflicts  of  interest. 

The  scope  of  the  peer  reviews  varies.  Some  reviews  are  strictly  limited  to  the  quality  of 
the  research,  and  to  assessment  of  the  quality  of  the  staff  and  facilities.  Other  reviews 
address  a  mixture  of  quality  and  relevance,  while  still  others  appear  limited  to  questions 
of  relevance.  Typically,  the  quality-oriented  reviews  employ  many  academics  who 
specialize  in  the  subjects  of  interest.  The  relevance  assessments  are  carried  out  by 
customers,  such  as  program  executive  officers  and  program  managers;  or  representatives 
of  soldiers,  such  as  senior  Anny  officers  from  the  Training  and  Doctrine  Command. 
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Most  of  the  Army’s  technical  peer  reviews  are  aimed  at  evaluating  individual  project 
areas.  A  few  look  more  broadly  at  the  laboratory’s  overall  efforts.  ARL’s  TAB  looks  at 
all  the  technical  work,  both  S&T-funded  and  customer  work;  the  Air  Force’s  SAB 
similarly  looks  at  all  the  AFRL  work.  An  important  question  is  whether  or  not  the  review 
extends  beyond  the  technical  work  itself  to  broader  questions,  such  as  the  status  of  staff, 
equipment,  facilities,  and  the  work  environment.  These  judgments  would  require  that  the 
panels  make  extensive  visits  to  the  laboratories,  talk  to  staff,  and  generally  familiarize 
themselves  with  the  environment.  . 

Most  of  the  panels  under  consideration  here  prepare  formal  written  reports  in  addition  to 
the  infonnal  feedback  provided  during  the  reviews.  The  reports  are  sent  to  managers  and 
are  available  to  the  research  staff.  In  some  cases  they  are  available  to  the  public.  The 
reports  prepared  for  the  DOE  weapons  laboratories  are  marked  FOUO.  In  some 
laboratories,  local  managers  are  required  to  report  back  the  actions  taken  in  response  to 
recommendations  made  by  the  panels. 
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Chapter  5:  Closing  Comments  &  Recommendations 


It  is  unlikely  that  any  one  model  of  peer  review  will  suffice  for  the  Army  laboratories. 
The  authors  believe,  however,  that  some  form  of  independent  external  review  of  technical 
quality  is  necessary  for  every  Army  laboratory.  The  Army  should  require  that  these 
reviews  be  undertaken,  even  if  differing  in  some  details  from  laboratory  to  laboratory. 

A  model  that  focuses  on  quality  review  of  the  technical  work  will  apply  well  to 
laboratories  that  perform  a  large  amount  of  basic  and  applied  research  (6.1  and  6.2). 
However,  if  a  laboratory  has  a  large  component  of  advanced  development  work  and 
beyond,  this  work  too  should  be  included  in  the  review.  In  such  cases,  the  panel  experts 
will  include  technically  trained  individuals  representing  both  customers  and  users.  Some 
of  the  development  and  engineering  centers  will  likely  want  to  devote  relatively  more 
effort  to  issues  of  project  relevance  to  the  armed  forces.  A  center  that  has  little  6.1 
research  will  probably  count  on  the  DRLM  basic  research  reviews.  However,  all  the 
technical  work,  whether  in  the  S&T  program  or  in  customer-sponsored  work,  should  be 
reviewed  for  quality.  This  can  be  combined  with  other  types  of  reviews,  or  evaluated 
separately. 

Table  1  contains  a  list  of  questions  relevant  to  the  design  of  the  peer  review  process; 
answers  to  these  questions  should  be  useful  for  comparing  and  contrasting  practices  at 
various  laboratories. 

Table  1:  Important  Questions  to  Consider  in  Peer  Review 


Is  there  any  peer  review  at  all? 

Who  manages  the  review? 

Are  external  experts  used  in  the  review? 

Are  the  experts  independent  and  free  of  conflicts  of  interest? 

Are  the  experts  external  to  the  project  staff,  or  external  to  the  laboratory? 
Are  the  experts  external  to  the  Army? 

Does  the  review  focus  on  issues  of  quality,  relevance,  or  both? 

Does  the  review  focus  on  individual  projects,  or  on  the  laboratory  as  a 
whole? 

Does  the  review  consider  the  quality  of  staff,  equipment,  facilities,  or  the 
working  environment? 

Are  explicit,  formal  reports  produced? 
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A  uniform  Army  definition  of  the  tenn  external  review  should  be  established.  A 
distinction  should  also  be  made  in  Army  tenninology  between  quality  peer  review  and 
relevance  review.  Whether  these  reviews  should  be  done  separately  or  in  combination  is 
an  open  question  that  may  be  answered  differently  in  different  laboratories.  Similarly,  the 
expertise  required  of  reviewers  will  vary.  External  review  panels  should  be  balanced 
between  the  requirements  of  expertise  and  “distance,”  from  the  reviewed  parties  (to  avoid 
damaging  conflicts  of  interest).  Contracting  with  an  outside  entity,  such  as  the  NRC  or 
some  other  independent  contractor,  to  convene  review  panels  and  manage  the  process 
will  enhance  the  credibility  of  the  results.  Alternatively,  reviews  could  be  managed  by  an 
entirely  separate  internal  organization,  as  is  done  in  the  DRLM  Basic  Research  Reviews. 
(The  BRR  is  a  hybrid  approach  wherein  the  panels  are  external  experts  but  the  naming  of 
members  and  convening  of  the  panels  are  directed  by  the  DRLM.  The  important  point  is 
that  credibility  is  enhanced  by  independent  reviews. 

The  scope  of  reviews  should  cover  individual  projects  in  enough  detail  that  quality  of  the 
work  is  exposed.  If  a  single  panel  is  used  for  an  entire  laboratory,  the  sampling  of 
projects  and  presenters  will  be  less  likely  to  give  a  reliable  picture  than  if  multiple  panels 
are  used,  each  focusing  on  a  particular  set  of  disciplines  or  programs.  A  trade-off  must  be 
made  between  cost  and  the  depth  of  the  reviews. 

To  evaluate  not  only  the  specific  projects  that  are  presented  but  also  to  assess  the  research 
environment — staff,  management,  equipment,  and  facilities — a  panel  will  need  to  spend 
2-3  days  at  the  laboratory  to  allow  walking  around,  talking  to  staff,  and  getting  a  feel  for 
the  ambience  of  the  laboratory.  Simply  sitting  in  a  conference  room  will  only  give  an 
impression  of  the  projects  selected  for  review,  and  little  else. 

Review  panels  should  provide  informal  feedback  to  the  management  and  staff  at  the  time 
of  the  review,  but  should  also  submit  a  fonnal  written  report.  This  report  should  be 
provided  to  interested  parties  within  the  laboratory  and  to  managers  up  the  chain  of 
command.  Internal  and  external  organizations  charged  with  oversight  of  the  laboratory 
should  receive  the  reports.  Part  of  the  process  should  be  to  require  the  laboratory  to 
respond  to  panel  recommendations  at  the  next  review. 

Recommendations 

The  following  recommendations  are  offered  as  the  basis  for  a  directive  from  the  Deputy 
Assistant  Secretary  of  the  Army  for  Research  and  Technology  (DAS(RT))  to  laboratory 
managers.  Since  there  is  unlikely  to  be  a  single  mode  of  review,  the  recommended 
approach  is  to  establish  guidelines  that  will  set  a  minimum  performance  bar  that  is  to  be 
met  or  exceeded  by  all  Anny  S&T  laboratories. 

The  DAS(RT)  should,  in  consultation  with  laboratory  management,  issue  a  written  policy 
on  peer  review  covering  all  Army  S&T  laboratories.  The  policy  should  be  accompanied 
by  detailed  guidance  along  the  lines  of  the  following  recommendations: 

Recommendation  1.  The  policy  should  require  the  laboratories  to  empower  outside 
groups  to  convene  peer  review  panels  and  manage  the  review  process.  The  policy  should 
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point  out  the  enhanced  credibility  of  panels  that  are  so  convened.  These  arrangements 
will  normally  be  created  through  the  use  of  contracts.  The  reputation  of  the  contractor 
should  be  an  important  factor  in  the  selection  process.  Specific  steps  should  be  taken  to 
ensure  that  conflict  of  interest  on  the  part  of  the  proposed  experts  is  dealt  with  either  by 
disqualifying  the  candidate  or,  in  the  case  of  very  small  pools  of  available  experts, 
making  the  potential  conflict  known  in  advance  and  achieving  balance  by  selection  of  the 
other  members.  Laboratory  managers  may  suggest  panel  members  but  should  not  control 
the  appointments.  Membership  will  vary,  depending  on  whether  the  laboratory  elects  to 
have  separate  reviews  for  quality  and  relevance  or  consolidated  reviews. 

Recommendation  2.  The  policy  should  require  that  the  reviews  themselves  be  done  by 
panels  of  experts  external  to,  and  independen  t  of  the  laboratory.  For  review  of  technical 
quality,  the  panels  should  consist  of  researchers  in  the  same  fields  drawn  from  academia, 
industry,  research  institutes,  and  other  government  laboratories.  For  review  of  relevance, 
the  panels  should  have  both  technical  experts  and  representatives  of  the  user,  such  as 
program  executive  officers  and  program  managers  and  the  TRADOC.  Retired  Army 
officers  can  be  of  help  in  both  quality  and  relevance  reviews. 

Recommendation  3.  Reviews  should  be  done  every  2  or  3  years.  To  spread  out  the 
burden  of  handling  the  reviews,  the  annual  review  process  should  be  staggered  such  that 
any  one  area  is  only  reviewed  every  2  or  3  years. 

Recommendation  4.  The  reviews  should  cover  technical  details  at  the  project  level. 
Enough  time  should  be  allowed  for  the  panel  to  hear  many  different  project  presentations, 
selected  to  give  a  true  picture  of  the  laboratory’s  work.  One  way  to  provide  good 
coverage  of  the  laboratory  programs  is  to  have  separate  panels  for  each  major  segment  of 
the  laboratory  program.  The  panels  should  be  allowed  to  spend  2  or  more  days  in  review. 

Recommendation  5.  The  panels  should  also  assess  the  quality  of  the  staff  the 
management  environment,  the  equipment,  and  the  facilities.  This  can  be  done  by 
encouraging  panel  members  to  walk  through  the  laboratory  and  to  talk  to  staff. 

Recommendation  6.  The  panels  should  provide  feedback  to  the  laboratory  staff  and 
prepare  formal  written  reports.  Reports  should  be  submitted  to  the  laboratory 
management,  and  copies  provided  to  relevant  members  of  the  chain  of  command.  Copies 
of  the  reports  should  be  filed  in  the  DAS(RT)  office.  The  laboratories  should  be  required 
to  respond  to  panel  recommendations  or  critiques  at  the  next  meeting. 

If  this  set  of  recommendations  is  adopted  and  executed,  the  result  should  be  improved 
quality  of  the  Army’s  technical  work,  enhanced  reputation  of  the  laboratories,  and 
strengthened  support  of  the  laboratories  among  the  Army’s  senior  leaders,  as  well  as 
within  the  Administration  and  in  Congress. 
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Appendix  A:  Interviewees 


The  following  individuals  were  interviewed  for  this  report: 

•  Air  Force 

o  Air  Force  Research  Laboratory — Michael  Kuliasha,  Chief  Technologist 

•  Navy 

o  Naval  Research  Laboratory — John  Montgomery,  Technical  Director 


•  Army 

o  Army  Materiel  Command 

■  Research,  Development,  and  Engineering  Command — Gary  Martin, 
Deputy  to  the  Commanding  General 

•  Aviation  and  Missile  RDEC — William  McCorkle,  Technical 
Director 

•  Armaments  RDEC — Joe  Lannon,  Technical  Director 

•  Army  Research  Laboratory — Joe  Mait,  ST 

•  Army  Research  Office — Dave  Skatrud,  Director 

•  Communications  and  Electronics  RDEC — Art  Ballato,  ST 
(with  Geo.  Oliva) 

•  Edgewood  Chemical  and  Biological  Center — Rick  Decker, 
Technical  Director 

•  Natick  Soldier  RDEC — Marilyn  Freeman,  Technical  Director 

•  Tank  and  Automotive  RDEC — Grace  Bocheneck,  Technical 
Director 

o  Army  Research  Institute  of  the  Behavioral  and  Social  Sciences — Michelle 
Sams,  Director 

o  Engineering  Research  and  Development  Center — James  Houston,  Director 

o  Medical  Research  and  Materiel  Command — Frasier  Glenn,  Principal  Assistant 
to  the  Commanding  General  for  Research  and  Development 

•  Department  of  Energy 

o  Lawrence  Livermore  National  Laboratory — Lee  Younker,  Director, 

Operations  and  Assessment 

o  Los  Alamos  National  Laboratory — Terry  Wallace,  Principal  Deputy  Director 
for  Science 

o  Sandia  National  Laboratory — Joan  Woodard,  Deputy  Laboratory  Director  for 
Nuclear  Weapons  (with  Gary  Sanders) 
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